3D image synthesis from depth encoded source view

ABSTRACT

A method of generating a synthesized 3D image from a source 2D image for display on a 3D display device including mapping a position on the display device to a fractional view number, and determining a source pixel of the 2D image to represent the said fractional view number, wherein the source pixel is determined as a function of the depth of a pixel at the position.

CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] This application is based upon and claims the benefit of priorityunder 35 U.S.C. § 119 from the prior Australian Patent Application No.2002952874, filed Nov. 25, 2002, the entire contents of which areincorporated herein by reference.

FIELD OF THE INVENTION

[0002] The present invention is directed towards an improved techniquefor the generation of images for use with autostereoscopic displays. Inparticular, the present invention relates to an improved method ofproducing images derived from a 2D image and associated depth map.

BACKGROUND OF THE INVENTION

[0003] A number of autostereoscopic displays are starting to appear onthe market, some of which require multiple images in order to provide anautostereoscopic image, and enable the viewer to retain a stereoscopiceffect despite movement of their head.

[0004] Such displays generally require images comprising a number ofviews created from a number of laterally displaced cameras. Such viewscan be originated from real cameras, generated using computer graphicimage (CGI) techniques, or synthesized from a 2D image and an associateddepth map.

[0005] The synthesis of multiple images from a 2D image and anassociated depth map has previously been disclosed by the presentApplicants in Australian patents AUS 714759 (10884/97) and AUS 738692(16472/99) included here in full by reference.

[0006] In U.S. Pat. Nos. 6,118,584 and 6,064,424, included here in fullby reference, van Berkel describes an autostereoscopic display thatrequires seven views. In German patent PCT WO 01/56302 A1, included herein full by reference, Grasnik describes an autostereoscopic display thatrequires eight views.

[0007] To those skilled in the art such displays are known to requiremulti-views or integer multi-views in order to display autostereoscopicimages.

[0008] Given the commercial availability of such multi-view displaysthere is a corresponding requirement for suitable images or content.

[0009] There is thus a need for a more efficient technique fordeveloping high quality images for multi-view autostereoscopic displays.Ideally the technique will be generic and can be applied to any displayrequiring multiple views (e.g. lenticular, parallax barrier, wavelengthselective filter).

OBJECT OF THE INVENTION

[0010] It is therefore an object of the present invention to provide animproved method for the real time generation of images, suitable for usewith multi-view autostereoscopic displays.

SUMMARY OF THE INVENTION

[0011] With the above object in mind the present invention provides amethod of creating images suitable for use with a multi-viewautostereoscopic display including the steps of:

[0012] 1. A view calculation means for determining a fractional view fora given position of the synthesized 3D image.

[0013] 2. A source pixel calculation means for relating the calculatedview number to a position in the source image.

[0014] 3. A weighted average means for calculating a source pixel valuegiven a fractional source position.

[0015] In another aspect the present invention provides a method forgenerating a synthesized 3D image from a source image including thesteps of:

[0016] determining a fractional view number for each pixel of a pixelarray, and

[0017] referring to a depth map of said source image to determine alocation offset of each pixel.

BRIEF DESCRIPTION OF THE DRAWINGS

[0018]FIG. 1 illustrates the arrangement and number of views.

[0019]FIG. 2 illustrates the perspective shift between two exampleviews.

[0020]FIG. 3 illustrates part of the pixel array of the synthesized 3Dimage.

[0021]FIG. 4 illustrates the pixel array of the source viewcorresponding to the synthesized 3D image.

[0022]FIG. 5 illustrates the value array of the depth map correspondingto the source view.

[0023]FIG. 6 illustrates an example mapping between the pixel shift andthe view number.

DETAILED DESCRIPTION OF THE INVENTION

[0024] Whilst content can be obtained from a number of laterallydisplaced cameras this invention relates to the generation of imagessynthesized from a 2D image and an associated depth map.

[0025] In the preferred embodiment of the present invention the processof generating a synthesized 3D image includes two steps: namely themapping of a position on the 3D display device to a fractional viewnumber and the subsequent determination of a suitable source pixel torepresent the said fractional view number.

[0026] A “view” may be defined as the projection of a 3D object or sceneon to a 2D image. For the purposes of multiview displays views aregenerally converged on a point of focus and numbered from left to rightas shown in FIG. 1. The optimal configuration of views depends on thecharacteristics of the screen and the contents of the scene. Each viewis captured by some form of camera or capture device, whether physicalor virtual. FIG. 2 illustrates two views of the object in FIG. 1—theshift in perspective effectively encodes the 3D structure of the scene.

[0027] An alternative definition of a view is to consider the anglebetween two cameras and a fixed point in the scene. For example, in FIG.1 view 3 may be considered to be at an angle of 2 degrees from thefront-on view 4. The present invention is not restricted to a fixedconfiguration of views but is able to generate arbitrary fractionalviews from a source view and an associated depth map (as described inthe Applicants previous disclosure PCT/AU96/00820).

View Determination

[0028] The first step of generating a synthesized 3D image involvesdetermining a fractional view number for each pixel of the pixel array.FIG. 3 illustrates a small section 5 of a typical pixel array. The array5 has a number of cells or pixels each containing a color and intensityvalue. Overlying the pixel array is an optical component 6 which may bearranged at an angle 7 α to the pixel array.

[0029] A generic formula for determining the source view for any givenpixel of the array was previously described by Van Berkel as$N = {\frac{\left( {k + k_{offset} - {3l\quad \tan \quad \alpha}} \right){mod}\quad X}{X}N_{tot}}$

[0030] where k is horizontal pixel index

[0031] k_(offset) is horizontal shift of the lenticular lens arrayrelative to the pixel array

[0032] α is angle of lenticular lens array relative to the pixel array

[0033] X is the number of views per lenticule

[0034] N_(tot) is total number of views

[0035] N is the view number of each sub pixel k,l

[0036] Given characteristics of the screen such as the angle 7 betweenthe lenticular and the pixel array α as well as the pitch of thelenticules a source view (N) can be calculated for any position (k,l) inthe synthesized 3D image.

[0037] Although Van Berkel's formula describes view determination forlenticular displays it may also be applied to parallax barrier systems.The barrier will simply obscure some views. Similarly, the formula mayalso be applied to wavelength selective filter arrays in which case someviews will be filtered. Further although van Berkel teaches taking thenearest integer view (N) for the actual mapping the fractional resultfrom the formula may be used directly.

[0038] In practice, the optical component overlying the pixel arraygives rise to a repeating pattern which maps view numbers to the pixelarray. This pattern can be represented as a mask, which is repeatedlyapplied. This leads to a gain in efficiency as van Berkel's formula onlyneeds to be calculated once for each mask element.

View Generation

[0039] Once a fractional view, N, has been calculated for a given k,lposition in the pixel array of the synthesized 3D image the presentinvention provides an inventive means for generating a pixel for therequired view. This approach uses a depth map to determine a locationoffset of the pixel in a single source view. This offset effectivelysimulates a perspective transformation by introducing parallax.

[0040]FIG. 4 illustrates the pixels 8 of the source view correspondingto the synthesized pixel array in FIG. 3. Similarly, FIG. 5 illustratesan array of depth values 9 corresponding to the pixels of the sourceview. To calculate the position of the pixel in the source viewcorresponding to the required view, N the following formula is used:

k′=k+3(Z _(k,l) P _(shift)(N)P _(strength) −P _(offset))

[0041] where:

[0042] Z_(k,l) is the depth at position k,l (which varies between 0 and1)

[0043] P_(shift)(N) is a function relating the view N to a parallaxshift (−1 . . . 1)

[0044] P_(strength) controls the total parallax strength

[0045] P_(offset) is used to control the parallax offset (or falseparallax)

[0046] The function describing the relationship between a view N and theassociated parallax shift P_(shift)(N) depends on the opticalcharacteristics of the 3D screen in question.

[0047]FIG. 6 shows a realistic example of the P_(shift)(N) function fora seven view display. With reference to FIG. 1 it is apparent that theshift is greater at the extreme views (1 and 7) than in the central view(4). It should be noted that this formulation can be used to relateshift to fractional views. For example, in FIG. 3 view 3.127 can berelated to a shift of approximately 0.25. In practice the P_(shift)(N)function may not be linear—for example it may be desired to reduce theamount of shift towards the extreme views.

[0048] The calculated position k′, l′ may not be an integer, in whichcase the nearest position may be used. Alternatively, the nearestinteger source pixels may be averaged using a weighted averagecalculation. As an alternative to weighted average calculation, anystatistic of the closest pixels to k′ may be calculated.

[0049] The current invention improves on the prior art by removing theneed to completely generate multiple views and is thereforesignificantly more efficient.

[0050] For example, when synthesizing a 3D image using nine views the 3Dimage only contains {fraction (1/9)} of each view—the remaining{fraction (8/9)} are discarded. With the direct source and depthsynthesis technique we only generate the {fraction (1/9)} of each viewwe require, effectively reducing the amount of calculations required bynine. In general, for display with N views the current inventionrequires N times less calculations than the prior art.

[0051] A further aspect of the current invention improves on the priorart by providing a mechanism for generating arbitrary fractional views.The prior art teaches the generation of a synthesized 3D image from anumber of fixed views. The said views, are fixed at the point they arecaptured by a physical or virtual camera. The present invention includesa view generation means, which enables arbitrary views to be generatedduring image synthesis. This provides improved flexibility as the samedata can be used to configure the views for any multiview 3D display.

[0052] The current invention improves on the prior art by providing amore efficient means for synthesizing a 3D image. Traditionally, a 3Dimage is synthesized by first generating the required number of completeviews and then subsampling these views to form a composite 3D image. Forexample, in an autostereoscopic display requiring 24 views, 24individual images would have to be generated requiring significantcomputation and memory resources. The current invention avoids the needfor generating these views by synthesizing the image directly using asingle view and a depth map.

[0053] Whilst the method and apparatus of the present invention has beensummarized and explained by illustrative application it will beappreciated by those skilled in the art that many widely varyingembodiments and applications arc within the teaching and scope of thepresent invention, and that the examples presented herein are by way ofillustration only and should not be construed as limiting the scope ofthis invention.

1. A method of generating a synthesized 3D image from a source 2D imagefor display on a 3D display device including: mapping a position on saiddisplay device to a fractional view number; and determining a sourcepixel of said 2D image to represent said fractional view number; whereinsaid source pixel is determined as a function of a depth of a pixel atsaid position.
 2. The method as claimed in claim 1, wherein mapping saidposition to said fractional view number is determined by:$N = {\frac{\left( {k + k_{offset} - {3l\quad \tan \quad \alpha}} \right){mod}\quad X}{X}N_{tot}}$

where k is a horizontal pixel index k_(offset) is a horizontal shift ofa lenticular lens array relative to a pixel array α is an angle of thelenticular lens array relative to the pixel array X is a number of viewsper lenticule N_(tot) is a total number of views N is a view number ofeach sub pixel k,l.
 3. The method as claimed in claim 1, wherein saidsource pixel is determined by: k′=k+3(Z _(k,l) P _(shift)(N)P_(strength) −P _(offset)) where: Z_(k,l) is the depth at position k,lP_(shift)(N) is a function relating the view N to a parallax shiftP_(strength) controls the total parallax strength P_(offset) is used tocontrol a parallax offset.
 4. The method as claimed in claim 1, whereinsaid determining said source pixel includes using the nearest integerposition to a calculated position.
 5. The method as claimed in claim 1,wherein said determining said source pixel includes taking a weightedaverage pixel value from pixels closest to a calculated position.
 6. Themethod as claimed in claim 3, wherein k′ is rounded to the nearestinteger value.
 7. The method as claimed in claim 3, wherein a weightedaverage of pixels closest to k′ is calculated.
 8. A method of generatinga 3D image for display on a multi-view autostereoscopic display deviceincluding: receiving a source 2D image and depth map; determining afractional view number for a position on said display device; anddetermining a source pixel for display at said position as a function ofsaid fractional view number and a depth assigned to an original pixellocated at said position on said source 2D image.
 9. The method asclaimed in claim 8, wherein said fractional view number is determinedby:$N = {\frac{\left( {k + k_{offset} - {3l\quad \tan \quad \alpha}} \right){mod}\quad X}{X}N_{tot}}$

where k is a horizontal pixel index k_(offset) is a horizontal shift ofa lenticular lens array relative to a pixel array α is an angle of thelenticular lens array relative to the pixel array X is a number of viewsper lenticule N_(tot) is a total number of views N is a view number ofeach sub pixel k,l.
 10. The method as claimed in claim 9, wherein saidsource pixel is determined by: k′=k+3(Z _(k,l) P _(shift)(N)P_(strength) −P _(offset)) where: Z_(k,l) is the depth at position k,lP_(shift)(N) is a function relating the view N to a parallax shiftP_(strength) controls a total parallax strength P_(offset) is used tocontrol a parallax offset.
 11. The method as claimed in claim 10,wherein k′ is rounded to the nearest integer value.
 12. The method asclaimed in claim 10, wherein a weighted average of pixels closest to k′is calculated.
 13. A method of generating a synthesized 3D image fordisplay on a 3D display device including: receiving a source 2D image,depth map, and a fractional view array; for a position on said displaydevice, determining an offset location for a pixel based on said depthmap and said fractional view; and mapping a pixel at said offsetlocation to said position to generate said synthesized 3D image.
 14. Themethod as claimed in claim 13, wherein said offset location isdetermined by: k′=k+3(Z _(k,l) P _(shift)(N)P _(strength) −P _(offset))where: Z_(k,l) is a depth at position k,l P_(shift) (N) is a functionrelating a view N to a parallax shift P_(strength) controls a totalparallax strength P_(offset) is used to control a parallax offset. 15.The method as claimed in claim 13, wherein said determining said offsetlocation includes using the nearest integer position to a calculatedposition.
 16. The method as claimed in claim 13, wherein saiddetermining said offset location includes taking a weighted averagepixel value from pixels closest to a calculated position.
 17. The methodas claimed in claim 14, wherein k′ is rounded to the nearest integervalue.
 18. The method as claimed in claim 14, wherein a weighted averageof pixels closest to k′ is calculated.
 19. A system for creating imagessuitable for display on a multi-view autostereoscopic display deviceincluding: a view calculation means to determine a fractional view for agiven position on said display device; a source pixel calculation meansfor mapping a source pixel to said position as a function of a depth ofa pixel at said position and said fractional view at said position. 20.The system as claimed in claim 19, wherein said view calculation meansdetermines said fractional view number by:$N = {\frac{\left( {k + k_{offset} - {3l\quad \tan \quad \alpha}} \right){mod}\quad X}{X}N_{tot}}$

where k is a horizontal pixel index k_(offset) is a horizontal shift ofa lenticular lens array relative to a pixel array α is an angle of thelenticular lens array relative to the pixel array X is a number of viewsper lenticule N_(tot) is q total number of views N is a view number ofeach sub pixel k,l.
 21. The system as claimed in claim 19, wherein saidsource pixel calculation means determines said source pixel by: k′=k+3(Z_(k,l) P _(shift)(N)P _(strength) −P _(offset)) where: Z_(k,l) is thedepth at position k,l P_(shift)(N) is a function relating a view N to aparallax shift P_(strength) controls a total parallax strengthP_(offset) is used to control a parallax offset.
 22. The system asclaimed in claim 21, wherein k′ is rounded to the nearest integer value.23. The system as claimed in claim 21, wherein a weighted average ofpixels closest to k′ is calculated.
 24. The system as claimed in claim21, wherein a statistic of pixels closest to K′ is calculated.
 25. Amethod as claimed in claim 3, wherein a statistic of pixels closest toK′ is calculated.
 26. The method as claimed in claim 10, wherein astatistic of pixels closest to K′ is calculated.
 27. The method asclaimed in claim 14, wherein a statistic of pixels closest to K′ iscalculated.